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DETAILED ACTION 



Requirement for Information Under 37 C.F.R. '1.105 



1 . Applicant and the assignee of this application are required under 37 CFR 1 .105 
to provide the following information that the examiner has determined is reasonably 
necessary to the examination of this application. 

2. The information is required to identify products and/or services embodying the 
disclosed subject matter of evaluating intangible assets. The Examiner upon conducting 
a search for prior art, discovered a web site for ProGrid Ventures (a copy of the 
information is attached to this request). Included within the information at the web site is 
an article titled "Measuring Intangibles - The ProGrid Story" by C. W. Bowman which is 
dated 1999. The article, on page 8, discloses the development of ProGrid tools over the 
past eight years. The article indicates that several tools were developed prior to the 
priority date of the instant application including ProGrid-RO from 1991-1993, Progrid-TA 
from 1993-1994, ProGrid-UN in 1994, and Progrid-CO in 1995. In response to this 
requirement please provide any known publications, brochures, manuals and press 
releases that describe the ProGrid products or services that were available or known to 
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the public prior to 1995 as described by the ProGrid web site and specifically the tools 
listed above that were developed from 1991 - 1995. 

3. The fee and certification requirements of 37 C.F.R. 1 .97 are waived for those 
documents submitted in reply to this requirement. This waiver extends only to those 
documents within the scope of this requirement under 37 C.F.R. 1 .105 that are 
included in the applicant=s first complete communication responding to this 
requirement. Any supplemental replies subsequent to the first communication 
responding to this requirement and any information disclosures beyond the scope of this 
requirement under 37 C.F.R. 1 .105 are subject to the fee and certification requirements 
of 37 C.F.R. 1.97. 

4. In responding to those requirements that require copies of documents, where the 
document is a bound text or a single article over 50 pages, the requirement may be met 
by providing copies of those pages that provide the particular subject matter indicated in 
the requirement, or where such subject matter is not indicated, the subject matter found 
in applicant=s disclosure. 

5. The applicant is reminded that the reply to this requirement must be made with 
candor and good faith under 37 CFR 1 .56. Where the applicant does not have or 
cannot readily obtain an item of required information, a statement that the item is 
unknown or cannot be readily obtained will be accepted as a complete response to the 
requirement for that item. 
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6, This requirement is subject to the provisions of 37 C.F.R. '1.134, 1.135 and 1.136 
and has a shortened statutory period of 2 months. EXTENSIONS OF THIS TIME 
PERIOD MAY BE GRANTED UNDER 37 CFR 1,1 36(a). 

Conclusion 

7. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Alexander Kalinowski, whose telephone number is (703) 
305-2398. The examiner can normally be reached on Monday to Thursday from 9:00 
AM to 6:30 PM. In addition, the examiner can be reached on alternate Fridays. 
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If any attempt to reached the examiner by telephone is unsuccessful, the 
examiner's supervisor, Joseph Thomas, can be reached on (703) 872-9306. The fax 
telephone number for this group is (703) 872-9306 (for official communications including 
After Final communications labeled Box AF). 

Hand delivered responses should be brought to Crystal Park 5, 2451 Crystal Drive, 
Arlington, VA, 7th Floor, receptionist. 




Alexander Kalinowski 



Primary Examiner 



Art Unit 3626 



2/6/2005 
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Measuring Intangibles 



Measuring intangibles can be considered to be an oxymoron. The 
Concise Oxford dictionary definition of an intangible is 
"that cannot be grasped mentally" 

Nevertheless, intangibles are frequently far more important to 
organizations than the tangible assets that we buy, buUd 
and use in daily life* 

This manuscript tells the ProGrid^ story, one attempt to put a 
measurement framework around 
"that Accountants are not able to count" 



C. (Clem) Bowman 

Decembers, 1999 
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Foreword 



ProGrid represents a paradigm shift in transforming organizations. It is recognized as the 
leading concept in tackling the difficult issue of the effective management of intangibles. 

The future value of an organization lies both in its codified intellectual capital and its advanced 
systems and processes for doing business - business with suppliers, customers, the public, 
governments and a myriad of other important stakeholders. The human capital of an 
organization represents a critical component of the intangible power of an organization. 
Unlike tangibles, intellectual assets do not depreciate over time but are normally only realized 
over the fulfillment of time. 

ProGrid pro\'ides the fimneworic for addressing the management of intangibles using unique 
tools that are customized to meet the values, priorities and expectations of individual 
organizations. 

The ProGrid system has evolved over a period of eight years in response to defined user needs. 
It has been developed by a core of experienced technology and business managers. The 
breadth of application has been greatly expanded by two dozen initial users who have added 
their own innovative concepts to the basic tools. Clem Bowman has captured the story of this 
evolution from initial conception to the current widespread application of the ProGrid system. 

What you will read in this manuscript is the tip of the iceberg. ProGrid is still evolving at a 
breathtaking pace. The excitement that has been generated over ProGrid will certainly 
continue as we enter the new millennium. We plan to issue periodic updates to this manuscript 
to demonstrate how ProGrid is being applied as a new thought process in capturing the 
intellectual power of organizations. 



J. CRod) McCulIough 



on behalf of 




Ventures Inc. 
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Chapter 1: The Importance of Intangibles 



1.1 Introduction 

Accountants have ruled the commercial world for most of the recent history of modem 
industrial development. This is not surprising in view of the adage "What cannot be 
measured, cannot be controlled". In the past, tangible assets were assumed to represent 
the majority of the wealth of companies. These were readily amenable to accurate 
assessment or "bean counting", to use a somewhat derogatory expression. The 
accountants' view of the world has been an important feature of national and 
international commerce. Their metrics have provided the framework for transactions 
among commercial enterprises. 

It has been realized in the last few decades that tangible assets were only one part of the 
wealth of a company or a nation. The book value of a company frequentfy has badly 
represented its strength, either by overlooking fiiture potential or, in some cases, 
exaggerating its value by assuming unrealistic longevity of its current manufacturing 
processes and products. 

Of the Fortune top 500 companies in the first listing in 1954, 360 would not appear in a 
similar listing of industrial companies today.' Would better knowledge of the strength of 
their intangible assets have predicted their demise? 

Intangible assets are difficult to measure. But many intangible things in life are 
intuitively measured and compared against a preset standard of performance. Selecting 
one's spouse is often used as an example. 

There is an important distinction between tangible and intangible assets. Tangible assets 
tend to have an intrinsic value independent of the obser\^er. The value of intangible 
assets, however, depends in large measure on the values, priorities and expectations of 
the observer. 

There is another distinctiorv a dimensional issue, one of time. Many tangible assets 
devalue over time. Intangible assets often realize their value only over time. This 
unfolding time dimension requires an examination of value over more than one 
dimension, as will be seen as our story unfolds. 

This document describes the evolution of a methodology to measure intangibles, called 
ProGrid®^. It is ''a work in progress" in the sense that current users are evolving it to fit a 
variety of organizational goals. 



' Forty Years of the 500, Carol J. Loomis, Kathleen C. Smyth, Suzanne BarI>Ti, Volume 131, No. 9, 40*^ 
Anniversary Issue, page 1 82 

^ PnoGrid* Trademark registered US # 2 J 76, 1 93, Canada # 487, 1 30 
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This Chapter 1 describes some of the features of intangible assets and the more precisely 
defined term of intellectual capital and presents the challenge of establishing a 
measurement framework. 

Chapter 2 outlines the very useful and relatively brief customized tools that are used to 
assist in making selections from altemative choices. Chapter 3 presents some of the 
principles and underpinnings of ProGrid and describes how the system has evolved to its 
current structure. Chapters 4 to 7 describe set-piece tools that are used for benchmarking 
and auditing purposes. Chapter 8 illustrates how the ProGrid methodology can be used 
outside the technical/scientific area, opening up unlimited possibilities for v^der scale 
application. 



1.2 Intellectual Capital 

The intangible assets of an organization are often referred to as intellectual capital. 
The components of intellectual capital are shown in the following chart^ 



Intellectual Capital 


Structural Capital 




Intellectual Assets 


Systems 


Human 






Processes 


Capital 


Intellectual 


Codified 


Relationships 




Property 


Knowledge 


Innovation 





Intellectual property is comprised of legally recognized knowledge, such as patents, 
trademarks, etc. Codified knowledge is knowledge that exists in a form that enables it to 
be shared with others, in the form of docunients, drawings, software, etc. Intellectual 
assets are the sum of intellectual property and codified knowledge. Structural capital is 
understood to comprise intellectual assets, plus the organization's systems, processes, key 
relationships (such as those with customers, business partners and suppliers), and 
innovative capacity. The components of structural capital are sometimes referred to as 
customer capital, organizational capital, process capital, innovation capital, etc. The sum 
of structural capital, and the knowledge, skills, capabilities, and motivation of its people 
(often referred to as human capital) constitutes an organization's intellectual capital. 

The following chart, developed by the Alberta Research Council"*, illustrates the dynamic 
relationships among the indi\adual components of intellectual capital. 



■ Source: Skandia, ICM Group, Saint-Onge, Matrix-Links etc. 
Alberta Research Council Annual Report, 1999 
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Is it possible to measure the value of each of these components of intellectual capital? Most 
organizations, in fact, still operate at the level of intellectual property. There are, however, a 
group of industry and government leaders who are active in developing both the defmitions of the 
other elements of intellectual capital and the frameworks within which these can be measured and 
tracked over time. 

Approaches for tackling this task are described below. 

13 Measuring Intellectual Capital- Market Value versus Book Value 

Skandia has suggested that any difference between the book value of a company's assets 
and its market value must be accounted for by the value of intellectual capital as shown in 
the following chart 
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There are theoretical and practical problems with this approach, however. Market value 
is affected by many factors, only some of which represent true intellectual capital. The 
above relationship may be a fairly good approximation for publicly-traded companies but 
less so for private companies, where the market value is unknown. It is also not valid 
when there is excessive "hype" such as for some of the Internet stocks. Nevertheless, it 
is one approach for which metrics can be applied, since both the book value and market 
value are known quantities for most publicly traded companies. 



1.4 Measuring Intellectual Capital- The ProGrid Approach 

Organizations that do not have an established reliable maricet value may still have 
substantial value in their stock of intellectual capital. In the ProGrid approach, three 
surrogates for market value can be defined which serve as predictors for future 
commercial success, Intellectual Assets, Practices and Processes, and Expected Impact. 
These factors become the key drivers for a matrix of performance criteria for the various 
ProGrid tools. 

ProGrid was initially developed as a benchmarking tool to assess the practices of research 
organizations in comparison to those of leading R&D organizations. It was later 
extended to the assessment of technologies arising from these organizations. In a later 
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stage of evolution, the ProGrid methodology was used to assess the capability of 
organizations to commercialize these technologies. 

The methodology has now been extended to a wide range of intangible assets, including 
selecting projects for funding by venture capital organizations. Proposals submitted to 
venture capitalists are usually submitted by early stage companies where physical assets 
are largely absent The evaluation of the future potential of these companies must be 
based almost entirely on the strength of their intangible assets, such as their technology 
and management skills. 

There are five key steps in applying ProGrid to the measurement of any intangible asset: 

Step 1- Define a series of criteria (e.g. the values, priorities and expectations of the 
measurer) with respect to which the intangible asset is to be measured. 

Step 2- Select two of these as independent overarching criteria that if both achieved to a 
high degree yield the highest possible value for the asset. Constmct an evaluation grid 
that has these two overarching criteria as the axes. 

Step 3- Organize the remaining criteria into a matrix, each cell of which represents a 
factor which if achieved contributes to either or both the overarching criteria. 

Step 4- Establish a series of quality or performance levels for each cell of the matrix. In 
ProGrid temiinology, these levels comprise the Language Ladder' '^* measurement system. 

Step 5- Construct a bar graph that compares the ratings of each of the matrix cells. 

By following the above steps, and undertaking a prescribed assessment procedure, the 
value of a specific intangible asset can be determined and compared to other assets. 

1.5 Building on the Literature 

Those involved in the development of ProGrid have experienced frustration with other 
evaluation methodologies and saw the need for a more rigorous and disciplined process 
than those used by many funding/investment organizations. Direct experience with a 
multi- million dollar resource investment fund left one of the developers with the belief 
that conventional approaches of collecting the opinions of knowledgeable experts did not 
provide the consistency needed to achieve optimum fimd performance. Continuing 
involvement of several of the developers in government peer reviews gave rise to 
concerns with respect to the time spent in undertaking the reviews and the difficulty in 
getting busy executives in academia and business to participate in a lengthy review 
process. This issue is discussed further in Chapter 3. 

With the incentive to search for an improved process, a number of currently used and 
historical evaluation processes were reviewed to learn from their practices. These 
included: 
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The Risk Management Matrix of the Boston Consulting Group 

This was a technique that used four-quadrant grids for displaying performance. The 
matrix uses Relative Market Share and Growth as the two axes. As noted in the review by 
Millet and Honton^, these terms represent cash generation and cash use respectively. 
Thus, these two axes are not independent variables. Further, Millet and Honton note that 
the portfolio analyses methods, of which the Boston Consulting Group Risk Management 
Matrix is an example, are "not only higjily judgemental but also highly arbitrary (even 
biased)". Such mefliods are qualitative and the axes are unsealed. The position of a 
technology in these matrices is not subject to quantitative examination by other 
evaluators. The ProGrid methodology was developed to overcome these deficiencies by 
using independent and scaled axes, relevant criteria and well-defined performance levels. 

The Kepner-Tregoe Decision Process^ 

This technique lays out key criteria for a selection process and uses a numerical scale to 
rank alternative choices. Thus methodology is particularly useful in problem solving by 
identifying the likely cause of an undesirable evoit by listing and weighting potential 
causes. This approach illustrates the importance of having clearly defined criteria and 
separating them into "musts" (overarching criteria in ProGrid terms) and "wants" 
(desired goals that cumulatively contribute to the final decision). 

The Blake Managerial Grid^ 

The Blake grid is used to optimize the balance between people- focused and production- 
focused human resource practices. This technique uses a grid that displays and compares 
different personnel behaviour practices. Although not quantitative, it shows how 
subjective factors can be organized into meaningful patterns. 

The Myers -Briggs Personality Test^ 

This evaluation test ascertains the personality traits of individuals by using carefully 
defined statements and randomized selections of preferred statements to define 
personality traits. It Ls useful in improving group dynamics by valuing different 
approaches to solving problems. This influenced the effort made in ProGrid to construct a 
calibrated scale of statements to establish comparative performance levels. 

ProGrid methodology, in part, is based on some of the proven and robust principles 
established by these well-known management techniques. 



A Manager's Guide To Technology Forecasting And Strategic Analysis Methods, 
Stephen M. Millet and Edward J. Honton, Battelle Prcss. 1991 
^ http:/Avww.kepner-tregoe/index.html 

^ The Managerial Grid," Robert R. Blake and Jane Mouton, Gulf Publishing Company, 

Library of Congress Number 64- 1 4724, 1 964 

^ http:/Avww.oise.on .ca/^^engel/coop.mbcareer.htm 
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1.6 The ProGrid Family 



TTie ProGrid family is a set of robust decision- assist methodologies within which there 
exists a set of tools which support each other, can be customized to meet the current 
needs of a very diverse set of users and continuously evolve to meet their future needs. 
Many of these relate to the commercialization of new concepts, new processes, new 
products and new services. From left to right in the following chart there is a shift fiom 
a technology focus to a more market or organizational focus. Proceeding down the chart 
there is a shift fiom screening tools to more defmitive and detailed evaluation tools. 



Tools for 
R&D Entities 



Tools for 
Assessing 
Technology 



ProGrid®-Quick 

First Screen for Project Selection 
— Technical Market 



I 



I 



Tools for Assessing 
Companies 
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I 



ProGrid®- 
RO/UN 

Best Practices 




ProGrid-TA® 

Technology/ 
Market Stiengths 




ProGrid®-CO/HR 

Organization/Physical/ 
Human Resources 


i 


> 


f 


1 






f 






ProGrid®-lC 





Some of these are set piece tools (e.g. ProGrid-RO and ProGrid-TA) that are used for 
benchmarking and auditing purposes. These have been employed in more than five 
countries and a sizable database has been accumulated. Other ProGrid tools have been 
customized for specific purposes, particularly those in the ProGrid- Quick category, and 
are used primarily as decisiorv assist tools. ProGrid- IC represents a collection of 
processes and tools for managing intellectual capital within an organization by 
establishing the current and desired fiature positions and tracking performance 
trajectories. These are customized for the needs of individual organizations. 



Descriptions of the members of the ProGrid family are described in the following 

chapters of this manuscript. As ProGrid evolves, its application outside the 

science/technology/commercial market fields has become apparent. One such application 
has been included. 



Copyright J 999 ProGrid Ventures hic, Clement W. Bowman Consulting Inc. 



7 



1.7 The ProGrid Evolutionary Path 



As development of ProGrid proceeds, it has increasingly been recognized as not just a 
collection of tools but an integrated system for managing the intellectual power of an 
organization. This evolution has taken place over the past eight years, with the follo^^'ing 
key milestones: 

1991 to 1993' Development of ProGrid- RO as a tool to address the relationship^ of an 
R&D organization to the "clients" it serves, and to the critical issue of balancing short 
and long terra programs, an issue recognized in every survey as critical to effective 
management of R&D, both public and private. 

1993 to 1994- Development of ProGrid-TA as a tool to manage the technology arising 
from R&D organizations, asseSwSing both the technology advantage and the market 
potential. 

1994- Development of ProGrid-UN in response to a University of Toronto strategic 
planning initiative to benchmark the university against North American leading 
institutions. 

1995- Development of ProGrid -CO to assess the ability of a company to sustain itself 
through continual innovation of its structure, practices and product offerings, 

1996 to current- Continued evolution of the practical applicatbn of the above tools 
through customization to meet individual client needs and the development of user- 
friendly eifective processes and software. 

1997- First major international benchmarking project, involving ProGrid- RO and the 
evolution of ProGrid-based intellectual capital measurement techniques. 

1999' Development of ProGrid- HR for perfomiance and compensation management, the 
first major application outside the direct technology/commercial arena. 

1999- Upward integration of the tools into timely, robust, consistent intelligent systems 
for the performance management of an entire organization. 

2000 onwards- Exploring the rest of the "iceberg'*. 
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Chapter 2: Customizing ProGrid (Pi oGrid Quick) 



2.1 Introduction 

ProGrid-Quick is a term applied to a wide variety of focused tools used to assist in 
making rapid highly effective decisions with strong stakeholder consensus. Examples of 
use include: 

• Selection of science fimding applications 

• Venture capital screening and selection 

• Project monitoring and control 

Funding/investing organizations that use the ProGrid methodology embed their values, 
priorities and expectations into an application form. These values, priorities and 
expectations are first assembled in the fomi of a matrix of performance factors and a set 
of calibrated performance levels (the Language Ladder^ ^ ) established for each factor. 
The application form includes a self-assessment section in wliich the Applicant selects 
perfomiance levels that represent the current status of the undertaking for which 
investment is being sought. As a key part of this process, the Applicant provides 
justification for the selections. The completed applications are normally submitted to a 
review group of up to eight Reviewers. The Reviewers will select their own ratings for 
the undertaking, based on the justifications provided by the Applicant and their own 
knowledge and experience. 

The Applicants and the Reviewers' assessments are input into the appropriate ProGrid 
software for analysis. The software will provide an automatic output report, in both chart 
and text form, containing the following information: 

• The grid position of the application with respect to the two overarching criteria 
established by the fiinding organization 

• The strengths and weaknesses of the application with respect to the matrix cells 
established as the assessment fi*amework (the profile bar chart) 

• A comparison of the responses among Reviewers 

• A sensitivity analysis of the assessment, identifying correctable weaknesses and 
potential fatal flaws 

• Specific comments of the Reviewers with respect to individual criteria or the 
application in total 

• A comparison of the ratings of the current application with those previously 
considered in the user's database 

• A customized report to provide constructive feed-back to the Applicant 

The software also has provision for: 

• Maintaining separate records for different themes within a funding program 

• Tracking stages in the course of development of a project. 
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The management team respoasible for making funding/investment decisions uses the 
output charts and text as an assist in making these decisions. The above process is 
illustrated in the following pictorial representation. 



Customizing a ProGrid Evaluation System 




Definition of values^ priorities and expectations 



Values, priorities and expectations 
expressed in ^Xanguage Ladder" format 





"Busy" reviewers 
validate assessment 




Applicants carry out self- 
assessment 




Management team uses 
ProGrid charts for real 
time decision-making 
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2.2 The Decision Process 

A more rigorous d^cription of the decision process is illustrated in the following flow 
chart. 



Values, Priorities and Expectations of Funding Organization 

Embedded in Matrix 



Language Laddei^^ 

Based on Cells of the 
Matrix 



Self- Assessment 

Applicant Provides 
Justification 



Reviewer 
Assessment 

Using the Same 
Form 



Data Display 

Data Digitized 
and Displayed 
Graphically 



Executive Body Use Data, Charts and Analyses as a Decision Aid 



Second and Later Round 
Financing 

Language Ladder Modified to Emphasize 
Progress 



Measuring Outcomes 

Quantitative and Qualitative Outcomes 
Based on Impact Cells firom Original Matrix 



2.3 The Evaluation Matrix 

Every customized ProGrid-Quick tool has a distinctive evaluation matrix, reflecting the 
values, priorities and expectations of the funding/investment organization. However, the 
same general factors appear in most matrices, and therefore there is a recognizable 
pattern among many of the matrix cells. Thus, it is possible to constmct a list of cells that 
have similar characteristics, as shown in the following generic matrix. 

There is a specific structure that has been found to be effective in developing the matrix. 
Columns A and C should group criteria that mainly influence one of the overarching 
criteria. Column B should contain criteria that influence both of the overarching criteria. 
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The 20 cells shown are typical of those that have been ased to date. Normally only seven 
to twelve are employed in a specific customized tool For the assessment of proposals 
where scientific merit and the qualifications of the principal investigators are the major 
selection criteria, the cells will mainly be those shown the first two columns, which, in 
turn, would be developed in a three column format (e.g. the impact for some users will be 
the connector for others). For the assessment of proposals where market impact is the 
ultimate goal, the cells in the right-hand column would have increased importance. 



Candidate Cells for Evaluation Matrices 





ZMtS The C&nn^cto&V: ■ yiiil^ 


The Market/Impacts [ 


The Advance 


HQP/Skills 


Technology Transfer 


The Team 


Collaboration 


Market Impact 


Facilities 


Industry Involvement 


Competition 


Capacity 


Intellectual Property 


Rate of Return 


Project Management 


Commercial Readiness 


Strategic Fit 


Business Plan 


Receptor Capacity 


Societal Impact 


Validation 


Financing 





2.4 Language Ladder Statements 

There are many versions of Language Ladder statement sets that apply for any of the 
cells in the above-described matrices. The specific language used reflects the values, 
priorities and expectations of the funding/investment organization. A compendium of the 
statement sets that have been xised to date has been accumulated and help guide the 
development of new customized tools. 

The following is a typical Language Ladder statement set for the criteria on "Validation" 
of a new concept: 

A. The concept has had a limited level of testing. 

B. Key elements of the concept have been confimied but an integrated system, model or 
prototype has not yet been developed. 

C. An integrated system, model or prototype has been developed and has been tested. 

D. An integrated system, model, or prototype has been developed and the 
product/process/service successfully tested under a fiill range of commercially relevant 
conditions. 

These statements should be constructed to be clearly distinguishable and as uniformly 
separated as possible. The progress fi*om "limited testing" to an "integrated prototype 
tested under commercial conditions" takes place over four well-defined ieps, that should 
be justifiable by the Applicant and readily verified by Reviewers. 



12 



Copyright 1999 ProGrid Ventures Inc, <Sc Clement W. Bowman Consulting Inc. 



2*5 Application Form and Process 

The application consists of two parts: 

♦ Applicant infomiation and a synopsis of the research/product/process/service. 

• A self- assessment by the Applicant with rCwSpect to specific performance factors. 

The application form is developed by the fiinding/investing organization and reflects the 
organization's values, priorities and expectations. The self-assessment portion of the 
application form consists of up to twelve performance factors. 

Organizations that use the ProGrid evaluation methodology will normally establish a 
committee or panel to review each application and make either the final decision or a 
recommendation to be coasidered by a higher- level body. In some cases, the committee 
will consist of the group who undertook the original reviews. In other cases, the 
committee may be composed entirely of individuals who have not been involved in the 
review process. It is desirable that the committee receives the ProGrid output reports in 
advance of the committee meeting. In many cases, there will be considerably more 
requests for fiinding than fiinds available. In other cases, funds may not be a limiting 
factor This should be established early in the meeting and a protocol developed 
dependent on this factor. 

Some competitions have only a small number of applications and the Committees 
involved may decide to review each application in either the order submitted or some 
other arbitrary order. The Chairperson will nomially develop a protocol compatible with 
the number and nature of applications. 

If funds are limiting, an effective technique is to use either or both the positioning grid, or 
an R- value rating table (the R- value is the ntasure of the distance to travel to the upper 
right hand comer of the positioning grid), to divide the applications into three groups: 
Group 1- Highly Rated Applications 
Group 2- Applications that border on the budgetary cut-point 
Group 3- Applications that are significantly below the budgetary cut point 

There is no single method for making the final decisions/recommendations, but the 
following is one approach where the number of applications is large and where funds are 
limiting. 

Group 1 applications are considered in order of the Rvalue ranking, determining if there 
are any significant differences among the Reviewers in any performance factor. If not, 
budgetary or other concerns of any Committee member should be raised and considered 
After this process, a consensus should be sought about funding. 

Group 2 applications may require more effort. As for Group 1, differences among 
Reviewers should be identified and discussed. In addition, the sensitivity (potential for 
future changes) in the project should be assessed. The likelihood that current weaknesses 
can be readily corrected should be addressed. Any fatal flaws should be identified. (The 
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sensitivity analysis described in the following section is useful for this purpose). After 
this discussion, it should be feasible to either confirm the current relative position of the 
application or to shift it upwards or downward in the seriatim. 

Group 3 applications can be considered in an open discussion, in which a Reviewer may 
challenge tfie weak position and present views contrary to the average Reviewer 
assessment. Examples that fall in this category would be Applications that have singular 
strengths that would warrant fiinding on a "prospective" basis. 

If fiinds are not limiting, the grid position, or R- value, of each application can still be 
used to obtain the relative strength of each application. It is helpfiil if there is a database 
of past applications to serve as a comparison framework. Over time, a ProGrid user will 
develop norms of the characteristics of applications that are effective predictors of future 
success. 



2.6 The Charts 



The Committee will have ax^ailable a series of standard ProGrid charts to assist in their 
review of the submitted applications. The following chart provides the current position 
on a grid with the overarching performance criteria that were selected by the funding 
organization as the two axes. This chart displays the spread among the assessments 
carried out by the Reviewers and the position as determined by the Applicant 
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The Reviewers have selected the performance fectors that best define the current position 
of the Application. In some cases, there is an expectation that the position will improve as 
development proceeds. Some of ftie improvements are largely within the control of the 
Applicant and would be key factors to address in the continuing development program. 
Other factors are largely outside the control of tiie Applicant and reflect external 
conditions. Similarly, there is the potential for a slippage in he current position if some 
of the controllable development expectations are not successful or if some of the external 
factors adversely affect future prospects. 

The ProGrid assessment methodology includes an optional expert system that predicts 
possible upsides and downsides based on the current position of the Application and 
experience with similar applications. The potential upsides of this project are categorized 
as "Project Upside" and "External Upside". For plotting purposes, the chart positions for 
the upsides are shown in the following chart as "Project Up" and "Max Up", with the 
latter reflecting the attainment of all the upsides. Similarly, the potential downsides are 
identified as "Project Down" and "Max Down". 
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The following bar chart shows a profile of strengtiis and weaknesses of the application 
uith respect to the perfonnance factors in the original ProGrid matrix. 
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The level of consensus among the Reviewers, and comparisons with the Applicants self- 
assessment for each performance factor, is provided in either bar chart or tabular form. 

Bar Chart: 
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Table Format: 





AppL 


Rev. 1 


Rev. 2 


Rev. 3 


Rev. 4 


Rev. 5 


Rev. 6 


Rev. 7 


Rev. 8 


Rating 
"The 
Project 
Team" 
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B 


B 


B 


B 


C 


B 







Over time, users wll accumulate a database of pre\aous applications and will learn 
through experience the characteristics of a successfiil project. This chart provides a 
comparison between the grid position of the current application with the grid position of 
all previously considered applications. 
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Criteria 2 



The software provides considerable flexibility in analysing the data in the database. 
Since this tends to be user specific, it is best to do this in a separate post-competition 
analysis. A table is provided in the ProGrid procedure with each Application on a 
separate line. This table can be sorted by R value (the distance from a data point to the 
upper comer of the grid, with (10, 10) expressed as 100% and (0, 0) expressed as 0%, 
with a running total of the cumulative dollars to that point in the table. 

Provision has also been included to permit the sorting by one performance factor only, or 
by a sub- set of performance factors. 
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If the Applications are divided into several phases, stages or themes, the average result 
for each group, and other statistical parameters, can be calculated and graphically 
displayed. If these analyses become routine, they can be built into the software as 
automatic processing procedures. 

2.7 Peer Review and ProGrid 

Organizations who provide funds for scientific projects in a formal competition typically 
use a Peer Review process to select winning projects. This is a time-honoured process 
that has served the scientific community well in the past and will continue to be a 
mainstay of project selection. Applications normally will consist of detailed descriptions 
of the research planned in rigorous detail Copies of papers describing previous related 
research will fiiequently be included along with fiiU curriculum vitae of the principal 
investigators. The documentation would be equivalent to a complete business plan for a 
commercial venture. Those involved in the review process are normally other 
researchers with experience in the area involved. For most of the reviewers, participating 
in such reviews is part of the normally accepted woricload, and it is feasible for each of 
them to commit several days to a formalized review and selection process. 

Peer review as described above has limitations in certain situations. When the volume of 
applications is large (several hundred or more), when there is a near-term business 
implication requiring a wider spectrum of experience of the reviewers, when there is a 
need for a high level of transparency in the review process and when a more consistent 
basis of providing feedback to apphcants is required (successfiil and unsuccessfiil), the 
standard peer review process requires modification. 

ProGrid is a form of peer review in that selected reviewers assess a submitted application 
form. However, the high focus of the applications on well-defined criteria selected by the 
fiinding body, the self- assessment by the Applicants, and the ease of validation by 
reviewers leads to a review process that meets the requirements of competitions where 
conventional peer review has been found to be less suitable. 

Organizations that have used ProGrid have reported the following advantages: 

• Comprehensive- ensures that all key aspects have been addressed (open 
architecture) 

• Achieves consensus- provides input fix)m all stakeholders 

• Rapid- busy experts are willing to participate in review process 

• Disciplined and reproducible 

• Graphical and easily understood 

• Ranks a large number of assets in a portfolio 

• Provides a corporate data base and memory 

• A self- learning methodology, constandy impro\dng 

• Focuses on issues key to making difficult decisions 
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Chapter 3: Principles of ProGrid 



3.1 The Importance of Establishing Assessment Criteria 

Great emphasis has been placed in this manuscript on the need for a clear definition of 
the criteria and values underlying the assessment process. When assessing tangible assets, 
the criteria are intrinsic to the asset and the value tends to steadily decay over time. For 
intangible assets, the criteria are more subjective and the value may grow in time. It is 
necessary to dig deeply into the vision, mission, goals and objectives of the organization 
for which the assessment is being carried out to detennine the basis for the assessment. 

Thus there is a shift in emphasis from the intrinsic value of the asset to the values of the 
person/organization interested in the asset. Value is in the "eyes of the beholder". 

How many criteria are required to represent the value equation? For the customized 
ProGrid tools discussed in the previous chapter for making fiinding/investment decisions, 
seven to twelve criteria have generaUy been found to be sufficient. For the set piece 
ProGrid tools discussed in the follovidng chapters, from 30 to 60 criteria have been found 
to be required to properly represent the assessment target. 

3.2 The Structure of the Evaluation Matrix 

For the set-piece tools, the individual criteria should be grouped to form the cells of a 
matrix. This is done both to reduce the complexity of the evaluation to a smaller number 
of key factors, but also to group criteria that have a common relationship. For the 
customized shorter tools, the individual criteria can be arranged in matrix fomi. 

How many cells are appropriate in the matrix? Trial and error has led to the conclusion 
that six to twelve cells can do the jab, with nine being a frequent occurrence. The three 
by three nine-cell matrix is a convenient format that allows three factors to be considered 
in each column of the matrix. The three-column format enables the separation of the 
matrix into the first and third columns representing the two overarching criteria and the 
middle column representing the connecting factors that link the overarching criteria. 
Funding organizations whose responsibilities end once the selection decision is made 
tend to use a fewer number of cells; venture capital companies normally use a higher 
number of cells to assist in tracking future progress. 

Several users have addressed the issue of whether a single matrix can be used either to 
span a broad area of interest (from more scientifically focused applications to more 
commercially directed proposals) or to span a period of time over which projects have 
matured. In some cases, it has been decided to use more that one matrix to span the 
focus/time spectmm involved. Although this has some advantages, it reduces the ability 
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to track projects as they mature. If the overriding mission of the funding/investment 
organization is to meet more advanced criteria in the future, it is preferable that these 
criteria be included in flie initial matrix, even though applications would not be expected 
initially to rate hi^ in those criteria. 

3.3 The Need for Two Overarching Criteria 

The question is frequently asked why a two-dimensional grid is used in ProGrid rather 
than a three or larger dimensional space. Many users have started the design process with 
three key overarching criteria. To date, experience has shown that one of the three 
criteria can be resolved into the other two, in effect becoming the middle column of the 
matrix. Is there something fimdamental about two dimensions? In many activities of life 
we are constantly bombarded with decisions that require a fast assessment of two choices. 
In many cases they represent short and long-term options, immediate gratification of 
needs or longer-term investment to meet fijture needs. 

Companies that fail due to the obsolescence of a dominant product have failed to provide 
for product replacement. They had been one -dimensional in their thinking. For 
indi\ndual staff members, keeping the inbox clean is an important short-term requirement 
in business life, but unless some long-term contributions are made to the organization, 
survival is unlikely. Thus, one of the axes in most of the ProGrid tools has a time 
dimension, in effect separating the current situation from an expected desired future. 

The taidency in life is generally to think in a one- dimensional mode. Extending this to 
two dimensions in which the second dimension provides a trajectory to the future is a 
major and worthy task. 

3.4 Independent Axes/Exclusion Zones 

It is essential that the grid axes be relatively independent. This means that the 
overarching criteria that are plotted on these axes can be achieved independently of each 
other Examination of the ProGrid matrices shown in this manuscript would intuitively 
suggest that this is tme since Columns A and C in general contribute to one axis only. 
The degree of **independency" dictates how much of the grid is accessible for plotting 
purposes. The percentage of the chart that is accessible depends on both the rating and 
weighting factors used in the ProGrid software. 

The following chart illustrates a typical case where there are two exclusion zones that are 
not accessible. The weighting and rating factors used in the methodology define these 
zones. It is important that these zones are not excessively large. If they are, it means that 
there are too many criteria that influence both axes. If carried to the limit, only the 
diagonal is accessible and in this case, the axes are no longer independent. The example 
shown represents a realistic case in which the two exclusion zones in the upper left comer 
and the lower right comer are reasonably small. Once a sizable database has been 
established, a multi- variate statistical analysis (e.g. Principal Components Analysis) can 
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be carried out to identify two independent axis variables that have maximum variance 
with respect to the data set. 
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3*5 Importance of Self-Assessment 

Most of the active ProGrid tools include self- assessment as a fundamental part of the 
process. In benchmarking, this is particularly important in that the purpose of 
benchmarking is to improve the performance of an organization. It is critical that the 
organization be actively involved in the process, since it will have the prime 
responsibility for acting on the results. 

Self- assessment is also nonnally used in the shorter ProGrid-Quick tools used as decision 
assists. It is the justification that the Applicant uses for the self-assessment that enables 
reviewers to rapidly validate the assessment. The clarity in which the Applicant is able to 
make his/her case is an important input into the reviewers' recommendations. 

Nevertheless, self- assessment is not a prerequisite for using ProGrid methodology. In 
certain situations, it may prove to be appropriate to exclude self-assessment, or at least 
not to disclose the self- assessment to re\aewers. In such cases, it will be important to 
ensure there is sufficient infonnation provided in the application form for the reviewers 
to make their independent judgements. Care should be taken not to fall back on the 
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traditional approaches of providing masses of information that reviewers have to dig into 
to extract the key fectors on which to base their decisions. 

3-6 Sensitivity Analyses 



Although the prime objective of ProGrid evaluations is to assess the current position of 
an intangible on the assessment grid, it has been usefid to build in a future forecast in 
several of the tools. The concept behind the forecast is the identification of the likely or 
preferred future state, and then a detemunation of what events must occur to bring the 
forecast to fruition. 



This approach can be visualized by examining the "space" between the steps in the four- 
step ProGrid Language Ladder, as shovm below; 



•Focus on the space 
between the steps of 
the ladder 
•Using the current 
position as a base, 
identify the tasks 
required to achieve a 
desired future state 



D toC 



C toB 



CtoD 




B toC 



AtoC 
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The emphasis flius shifts from a static assessment of current position to ftie more dynamic 
analysis of what is needed to either move up the Language Ladder and/or what events 
need to be watched to avoid "falling off the ladder". 

As an example, consider the following Language Ladder statement set. 

With respect to the products/processes/services provided by this concept: 

A. there is no evidence at this time of customer acceptance. 

B. preliminary maricet studies indicate positive customer acceptance. 

C. specific customers have been identified who have indicated their intention to 
place orders. 

D. current or new customers have given firm orders today. 

A funding/investment organization could decide that the A level represents a fatal flaw. 
If there is no evidence of customer acceptance at this time, the concept is too ill defined 
or early stage to warrant support. 

However, there is nothing implicit in statements B and C that would preempt reaching a 
higher level in the ladder. It is relatively easy to define an action phrase to describe what 
needs to occur to achieve a higher level. 

The B to C shift phrase would require that market studies be extended and specific 
customers be identified who have indicated their intent to place orders. A shift to the D 
level would add the concept of submission of firm orders. 

On the downside, a D to C shift could occur if the customers who have placed orders 
decide to withdraw them, for whatever reason. 

Thus, it is feasible to develop an expert system that will "hard-wire" a number of likely 
future events and predict both a future outcome and the events needed to attain this 
outcome. The expert system can be designed to identify key actions and milestones b be 
used in the execution of the project and for tracking progress from conception to 
completion. 

3.7 Why Four Steps in the Language Ladder? 

The selection of four steps in the Language Ladder came about as a result of considerable 
trial and error. One powerful assessment methodology that has been applied to the safety 
of laboratories uses a two-step ladder- a laboratory is either safe or not safe with respect 
to a large number of criteria. This binary approach works well for a situation where 
success can be precisely defined However, for intangibles in genera], success is often 
time dependent and involves some form of evolutionary patii 

The first ProGrid tools used a ten- step ladder with only the top rung being defmed. 
Progress toward the top rung was expressed numerically in numbers ranging from 1 to 
10. Evaluators found it necessary to mentally construct their own plateaus of value and 
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suggested that it would be more effective if some of the intemnediate levels were defined 
and agreed to among a group of evaluators. 

The next attempt was the development of a three- step ladder that provided two steps 
below the top rung. There was a tendency for re\aewers to migrate to the middle rung 
and the level of differentiation among intangibles was frequently small. Extending the 
Language Ladder to four steps forces the evaluators to select above and below average 
values. 

Although there has been some discussion about adding further mngs to the ladder, most 
users have found that the four- step ladder provides a sufficient level of diffo^ntiation. 



3.8 ProGrid Software 

No over\aew of ProGrid would be complete without a discussion of the softv^^are used to 
produce the charts and graphs showTi in this manuscript. 

ProGrid is neutral with respect to the calculation and graphing programs. However, the 
ready access of most users to Excel spreadsheets and graphics and to the ACCESS 
database program has meant that these have been used to generate the majority of 
currently used ProGrid systems. 

Initially, the calculations and graphics were produced by Excel alone, and reports 
produced manually using a word processor. This combination worked well for the initial 
set- piece tools and for the customized ProGrid -Quicks up to about fifty applications per 
competition. 

The subsequent evolution of software development has taken the following paths: 
Set-Piece Tools 

ProGrid-TA (discussed in Chapter 4) was the first tool to experience logjams in 
processing. Over 200 technology/markets were assessed before it was realized tfie time 
spent in producing a comprehensive report of the findings was going to restrict 
commercial use. Skilled report writers would take at least a day to analyze the 
interactions among the 37 statements sets and to produce a realistic report of the current 
position of the technology/market pair and its fixture prospects. 

An expert system in Excel was developed by experienced technology managers that 
replicated the evaluation protocol that previously had been done manually. The software 
generated the report immediately on conclusion of the ProGrid-TA assessment session. 
Some users of ProGrid-TA have since further customized the report to meet specific 
needs. 

The final step in the evolution of ProGrid-TA sofKvare was to insert an ACCESS 
database front-end that provided for data entry and data storage. The end result has been 
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a three-component software system, an ACCESS fix)nt-end for data entry, an EXCEL 
calculation module for carrying out the ProGrid calculations, and a linked Excel report 
module that produces the output report. 

Customized Tools 

Some users of customized tools receive a very large number of applications in a 
competition, measured in the hundreds. The time spent in opening, naming, inserting 
data into, and saving the necessary files was onerous and became a serious deterrent to 
using the ProGrid system. This was overcome by adding an ACCESS database 
management system to the front- end of the Excel calculation program. The ACCESS 
database provided for data entry and file management, automatically opening, naming 
and saving individual files. In a similar fashion to the ProGrid-TA report described 
above, an Excel output report was customized to fit individual user needs. 

As with ProGrid- TA, the end result has been a three component software system, an 
ACCESS front end for data entry, an EXCEL calculation module for carrying out the 
ProGrid calculations and a linked Excel report module that produces the output report. 

As the evolution continues, more of the "expert" part of the system is being transferred 
from Excel to ACCESS to pemiit more rapid customization of the tools. Customization 
of the output report has been a major part of ongoing upgrades. 
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Chapter 4: Technology Assessment (ProGrid-TA ) 



4.1 Introduction 

ProGrid-TA® is a powerfiil tool for assessing the technical and commercial readiness of 
technologies, for making decisions at critical gates in a technology's evolution, for 
following the course of a technology over time, and for prioritizing a portfolio of 
technologies. The mefliodology recognizes that future success depends on two relatively 
independent targets, technical strength and market strength. 

Development of the ProGrid-TA tool involved the standard five ProGrid steps: 

Step /- Define a series of evaluation criteria with respect to which the technology is to be 
measured. 

Step 2- Select two of these as independent overarching criteria that if both achieved to a 
high degree yield the highest possible value for the technology. Construct an evaluation 
grid that has the two overarching criteria as the axes. 

Step 3- Organize the remaining criteria into a matrix, each cell of which contains related 
criteria. 

Step 4 Establish a series of quality or performance levels for each cell of the matrix. In 
ProGrid temiinology, these levels comprise the Language Ladder™ measurement system. 

Step 5- Constmct a bar graph that compares the ratings for each of the matrix cells. 

4.2 Evaluation Criteria 

Discussions with various technology developers with experience in commercializing new 
concepts generated a large number of key criteria that are indicative of technical and 
commercial success. 



Over time, the lust has been modified to flie following 37 criteria: 



Scientific Basis 


Advance on Prior Art 


Uniqueness 


Pervasiveness 


Proof of Concept 


System Integration 


External Validation 


Technical Credentials of Project Team 


Understanding Market Needs 


Supportinj^ Technical Networks 



' ProGrid-TA® Trademark registered US # 1,942,770 Canada # 487,128 
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Current Stage of Development 


Level of Development Required to Confirm 
Commercial RecidinesK 




Patent Position for Cnnvt'iQht Pn^itinn) 








OC-C/i/C JyJf J ffllJf L/vCfflGflLo 






A vn idn w cp 


Dpjjpndpnrp nyj f^thpf 
Products/ Processe ^/Service v 


RohustneK^ 


Market Acceotance 


Market Impact 


Geographical Reach 


Competitors ' Strength 


Regulatory Compliance 


Regulatory Leadership 


Price/Margins 


Competitor 's Price Sensitivity 


Cost Reduction Opportunities 


Speed of Commercialization 


Partners 


Market Trials 


Marketing Networks 


Investment A vailabilitv 





4,3 Overarching Criteria 

Most of the above criteria will influence either or both the technical or commercial value 
of the technology under consideratioa This led to the selection of the following two 
overarching criteria: 

Technology Assessment 

Commercial Assessment 
as the axes of the evaluation grid as shown in the following chart. 
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4.4 The Evaluation Matrix 



The previously listed criteria can be grouped into clusters of related factors in the form of 
the following matrix. 



■Scientific Sti^ 




pliiiiiililliiiiiiilipfe 


Technical 
Framework 


Commercial 
Readiness 


Market 
Characteristics 


Level of 
Verification 


Proprietary 
Strength 


Margin and 
Profit Potential 


Excellence of 
Project Team 


Technological 
Durability 


Commercialization 
Channels 



The columns in this matrix represent a progression from factors that largely influence the 
scientific strength of the technology in Column A, to those that largely influence its 
commercial strength in Column C and those that are necessary factors to link the 
technology to the maricet place in Column B, serving as the connecting factors. 

4.5 The Language Ladder 

This is the critical step that establishes the metrics of the ProGrid process. Through 
experience, it has been found tfiat a set of four steps in the Language Ladder works well. 
The choice of an even number is important in that it reduces the tendency for evaluators 
who use the ProGrid methodology to migrate to the middle *'neutral" position. 

The following are two examples of Language Ladder statement sets that are used in ftie 
current ProGrid- TA procedure. 

Advance on Prior Art 

The concept represents: 

A a relatively small advance on the prior art that would not be apparent to most 
users. 

B. a definable and measurable extension of the prior art that will be discerned by 
discriminating users. 

C. a significant and readily recognizable improvement over the prior art, but the 
basic scientific and technical principles are similar. 

D. a major advance on the prior art and embodies significantly diflFerent principles. 
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Market Acceptance 

With respect to the products/processes/services provided by this concept: 

A. there is no evidence at this time of customer acceptance. 

B. preliminary market studies indicate positive customer acceptance. 

C. specific customers have been identified who have indicated their intention to 
place orders. 

D. current or new customers have given firm orders today. 



4.6 The Evaluation Grid 

Once a technical asset has been evaluated using the ProGrid methodology, its position 
can be represented on the following grid, depending on its state of scientific and 
commercial readiness. The data in this chart represent technology/market pairs that were 
evaluated to the end of 1994. 
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At the inception of a new idea, its location on this grid will likely be close to the origin. 
As the idea matures into a bona fide technology, it will follow a specific trajectory. If the 
technical and maricet developments proceed in parallel, the trajectory will follow a 45 
degree diagonal. If the market develops faster than the technology, or if the technical 
content is relatively low, the trajectory will lie closer to the x-axis. Conversely, if 
technical progress proceeds faster than market development, or if the technical content is 
very hi^, the trajectory will lie closer to the y-axis. Where the trajectory ends will 
determine the overall merits of the technology. The division of the upper right hand 
quadrant into four zones illustrates the characteristics of technologies that fall within 
those zones. The nature of technologies that lie in the stalled zone vAW be discussed later; 
the characteristics of the other zones are self- explanatory. 

The four- quadrant grid is useful for obtaining an overall view of the status of a 
technology. An alternative display of the status is shown in the following chart, in which 
the four- quadrant boundaries are replaced by three curves that are concentric with the 
point 10,10. Points that lie on one of these curves have different combinations of 
technical and commercial strength but have the same distance to * travel" to reach the 
upper right hand comer of the grid. With this display, zones between the cur\'es can be 
described as "Embryonic", "Emerging", "Developing*' and "Commercially Ready". 



ProGrid-TA Evaluations 




0.0 5.0 10.0 

Commercial Strength 
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A factor "R" can be defined as the distance from a data point to the upper comer of the 
grid, with (10, 10) expressed as 100% and (0, 0) expressed as 0%. 

The R factor can be used to rank a set of technologies, or any other intangible asset, as 
shown in the table below. Ten technologies that have diflFerent X, Y values and therefore 
different R- values are compared and sorted. 
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4.7 The Technology Profile 

A second and important output of the ProGrid methodology is the Technology Profile bar 
chart with the bars representing the strength of each cell of the matrix previously 
described. 

The following chart shows profiles for two technologies, one having strength on the left 
side of the chart (scientific strength) and the other having strength on the right side 
(market strength). 

Over 200 technology/market pairs have now been assessed using ProGrid-TA, and 
specific profiles have been identified and classified by descriptive names, such as: 

• Technology Push 

• Market- Pull 

• Stalled 

• Commercially Ready 

Many cases of a stalled technology have been encountered which have modest strength 
across the profile (in the 45 to 60% range), but no commanding strengths and no major 
weaknesses, i.e. a flat profile. This is typical of a mature technology that has been 
involved in several commercial launch attempts where the obvious weaknesses have been 
corrected but no outstanding strengths have emerged This is a dangerous combination. 
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The Technology Profile 




4.8 Advanced Features 

Once a current position for a technical asset is defined, it is possible to develop an expert 
system to predict the future position. 

Possible future rating values are determined by the following relationship: 

Future value = Current value (A, B, or C) + increment (x, y or z) 

Where x is the number of improvement steps which the asset is likely to achieve if its 
cunent position is at level A, >^ is the number of improvement steps that the asset is likely 
to achieve if its current position is at level B, and z is the number of improvement steps 
the asset is likely to achieve if its current position is at level C. The values jc, y z are 
detemiined by those wifli e:q)erience in the development and commercialization of 
technology. Once determined, tfie .r, y, z values are kept constant until experience shows 
a change is appropriate. 

Similarly, it is feasible to estimate what downsides might occur if certain events do not 
work out as expected. 
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Future value = Current value (B, C, or D) - increment (x, y or z) 

Where x is the number of deterioration steps which the asset is likely to experience if its 
current position is at level B, y is the number of deterioration steps that the asset is likely 
to experience if its current position is at level C, and z is the number of deterioration steps 
the asset is likely to experience if its current position is at level D. 

The individual criteria in ProGrid-TA can be grouped into various categories, such as 
those that affect the technical development program, project fundamentals, competitive 
position and market confmnation. Applying the sensiti\aty analysis to each category 
results in an S curve plot as shown in the following chart. The expert system can be 
constructed to produce a list of actions required to achieve the upsides and a list of areas 
to address to protect against the downsides. 

Undertaking the ProGrid assessment at periodic intervals will show the level of progress 
that has been made. 



Potential Future Positions 




0 5 10 

Commercial Assessment 



Copyright J999,ProGrid Ventures Inc. & Clement W, Bowman Consulting Inc. 



33 



4.9 The Assessment Process 



The ProGrid-TA technology assessment procedure is best carried out by a small panel of 
people knowledgeable about the technology/maricet to be assessed. This normally will 
include indi\dduals representing the technology developers, the commercialization team, 
and preferably one or two individuals who are not part of these two groups but whose 
participation will add credibility to the process. 

A trained facihtator adds considerable value to the assessment by ensuring that the 
assessment decisions are consistent and well supported. It is essential that agreement is 
reached regarding the specifics of the technology being assessed and the markets to be 
served. The facilitator provides documents to support the assessment process. 



4.10 Looking Beyond Technology/Markets (ProGrid-CS) 

Venture Capital companies have historically placed pre-eminent importance on the 
quality of the management team in start-up companies. They have learned to make 
accurate judgements about the capability of the commercializing team based on a few 
observations- essentially a "gut feeP' about the quality and commitment of the principals. 

The ProGrid-TA approach uses a different sequence regarding a new venture; if the 
technical assets are strong and a viable market exists, then the venture deserves closer 
examination regardless of the current strengths and weaknesses of the commercializing 
organization. TTiis approach provides the opportunity to add value by facilitating the 
addition of the missing components. If the ProGrid-TA assessment is encouraging, a 
companion tool called ProGrid-CS can be employed to determine the capacity of the 
commercialization organization and its strengths and weaknesses. 

Eleven criteria are used in the assessment, as shown in the following matrix. Column A 
includes the factors related to people resources, Column C factors related to the strength 
of the business and Column B the connecting factors that are needed to link the human 
resources to the business opportunity. 
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The example shown in the following two charts illustrate the situation of an organization 
with strong people skills but with gaps in organizational capacity, particularly in 
production and marketing and in the lack of a credible revenue plan. ProGrid-CS is a 
rapid first look at a company, which is more folly analysed in the tool described in 
Chapter 6, ProGrid-CO. 
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Chapter 5: Benchmarking R&P (ProGrid-RO) 



5.1 Introduction 

Many research organizations have evolved during the past thirty years fh)m organizations 
focused primarily on long-term research flowng fix)m relatively rigid missions to those 
that now have flexible mandates aligned to the changing needs of a variety of clients. 
The concept of performance measurement has changed dramatically from the time wiien 
scientific excellence was the primary criterion for success. This evolution has not been 
linear and has involved a number of different relationships between R&D organizations 
and the stakeholders served. This chapter traces the evolution through various stages, 
and relates these to the types of performance measurement and benchmarking practices 
that have been used. 

Meeting both the short and long-term expectations of stakeholders has repeatedly been 
identified as one of the major challenges facmg research managers. This issue can be 
represented as the balance between the use of existing intellectual capital to meet the 
current business needs of clients and the renewal of the bank of intellectual capital to 
ensure long-term survival. ProGrid-RO was developed to assess how well an R&D 
organization balances these competing demands. 

5.2 The Development of ProGrid-RO 

In the 1980s, the author undertook a series of cost/benefit analyses of various research 
programs.'^ The decision to undertake these analyses was in large part a response to 
outside pressures, frequently at times of orgahizational stress. A more systemic and 
rigorous methodology was developed in the early i990s involving a matrix of 
performance areas, ranging from those in complete control of the R&D organization to 
those involving interactions with the owner, clients and external stakeholders.' ' 

The next step was the development of a mathematical framework to more quantitatively 
identify the performance level of an organization, for tracking performance over time and 
for comparing similar organizations.^^ The method recognizes that there are frequently 
two major overarching criteria for successful organizations, equivalent to short-term and 
long-term performance. This led to the concept of a performance grid with tiiese two 
criteria as the axes. 



Research at Esso - Sixty Years of Practice - Organizing for the Future, C.W. Bowman, Submission for 
the 23 rd Conference of the Canadian Research Management Association, Jasper, Alberta, September 
22-25, 1985 

Measuring Good Research Management, Research &Techno logy Management, March-April 1992, 
Volume 35 No.2, p 13. 

*^ Benchmarking Research Performance, R, S. Jane Memorial Lecture, Canadian Society for Chemical 
Engineering Conference, Oct 3-6, 1993 • 
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Recognizing that short and long-term horizons are surrogate terms for the use and 
generation of intellectual capital, respectively, these latter terms are now used as the 
defined axes of the perfomiance grid. 

5.3 Evolving R&D Practices 

R&D organizations have evolved during the past three decades, reflecting in large 
measure the degree to which they align themselves with business cost centres. It is useful 
to define two "bookend" types of organizations to illustrate the changing relationships. 

Type X Organizations- Organizations whose mandate is determined solely by the 
current business interests of clients and who hire staff and select and monitor projects on 
this basis. Deliverables are closely defined and the technology is transferred under 
contractual relationships within a corporate envelope. Although eflFective at meeting the 
needs of clients, these " organizations frequently have difficulty adding to their intellectual 
capital. 

Type Y Organizations- Organizations whose mandate is defined by a scientific charter 
and who hire staff and select and monitor projects based on meeting the objectives of this 
charter. Specific deliverables are generally not defined and technical information is made 
available through publications to the broad scientific network. Although effective at 
creating intellectual capital, these organizations are usuaUy not involved in the 
commercial use of this capital. 

The evolution can be described on a positioning grid with the use of intellectual capital 
and the generation of intellectual capital employed as grid axes, as shown in the charts on 
the following page. 

Phase 1 - The Science Leader- Type Y Oi^anization (Chart a) 

Organizations in Phase 1 received essentially all their funding as a core "grant" fiom the 
owner, with only a general statement of expected deliverables. "Good science", supported 
by access to the world pool of science and technological information, was the driver for 
R&D programs. These organizations were generally successful in generating new 
intellectual capital, which frequently resulted in new processes, products or services 
having significant technological impact However, the transfer of technology to clients 
tended to be sporadic and dependent on the personal relationships developed by senior 
R&D management with counterparts in the business units. Central corporate research 
laboratories during the 1960s had strong Type Y characteristics. 

Phase 2 - Business Alignment- Type X Organization (Chart b) 

Many owners became dissatisfied with the impact of Phase 1 organizations, and moved to 
an arrangement whereby R&D contracts were provided directly from the business cost 
centres. The programs were approved by and in many cases abnost entirely defined by 
these centres. R&D projects were short-term and responded to the current needs of the 
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business, with limited attention to global technology developments. The benefits of R&D 
were determined through periodic audits of successful projects. These organizations, 
although effective at technology transfer, were less effective at replacing their depleting 
intellectual capacity. This situation was prevalent with many industrial research 
organizations during the 1970s. 



Chart a- The Science Leader 
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Chart b- Business Alignment 
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Chart c- Dual Missions 
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Chart d- Full Integration 
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Phase 3 - Dual Missions - Type X + Type Y Organization (Chart c) 

Owners generally felt more comfortable with Phase 2 results, but were concerned that 
there were opportunities and threats not being addressed by this focus on the short-term 
needs of the business units. Some owners reinstated a modest level of corporate funding 
for longer-term R&D, with the selection of projects largely residing with the R&D 
organization for that component of the total budget The business lines still provided the 
majority of tiie funding for the organization. The management process for the short and 
long-term missions were not integrated. Whether this dual mission approach was 
effective is debatable. It did provide research management some discretion to initiate 



38 



Copyright J 999 ProGrid Ventures Inc. & Clement W. Bowman Consulting Inc. 



exploratory programs, but the capacity to expand these when successful was limited. A 
number of industrial research organizations adopted this strategy in the 1 980s. 

Phase 4 - Full Integration- Type XA" Organization (Chart d) 

Most companies realized in the late 1980s that they were entering a new fiercely 
competitive global economy in which technology would become a key strategic weapon. 
It l>ecame imperative that technology be fiilly integrated into the corporate strategic plan. 
It is necessary to involve senior levels of both business and research management in this 
process and to focus on both the short and long-term technology needs of the corporation. 
An attempt has been made to illustrate how this level of integration might occur in the 
foOowing chart. This shows the development and commercialization of technology as an 
integrated system in which the owners establish strategies and allocate resources. 
Technology recipients are actively involved in the definition of program elements. The 
impact of the external world pool of technology is felt by all participants, not just the 
R&D organization. An aggressive R&D management unit is needed to energize this 
process. 



World Pool of Scientific and Technological Information 
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5.4 The ProGrid-RO Methodology 

ProGrid was developed to benchmaric how R&D organizations perfonned in each phase 
of development and to plot the route to Phase 4 where appropriate. 
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The problem with most evaluation approaches is that they tend to be qualitative in nature, 
introduced by external consultants and generally applied as one-time events. Hiere is a 
tendency to relapse into old practices once the enthusiasm for change has diminished. 
Unless the changes become an integral part of internal processes and embedded into the 
culture of the organization, there is a high risk of losing the gains that have been made. 
What is needed is a method that provides a mathematical framework for quantitatively 
measuring an "intangible" asset A desired attribute would be a methodology which itself 
learns and improves through experience. 

ProGrid-RO defines the current position of an organization with respect to the use of and 
the generation of intellectual capital. It has now been used in five countries for 30 
different organizations. A sizable database has been accumulated. 

The key outputs are "Direction*' and "Distance from Goaf. These lead to actions for any 
needed course correction and also provide an indication of how far an oiganization is 
from its own perception of "high performance". 

The ProGrid results are displayed in chart form. There is no single correct position on 
the charts- there is only the "right" position for an organization in reference to its 
mandate and mission. The p*ocedure is a self- assessment process and can be considered 
as a "self-satisfaction" survey. Once used, the procedure can then be adapted to track 
future progress. 



5.5 Setting the Direction- Module 1 



Through the application of Module 1, a trajectory for an organization is determined on a 
grid that has Type X and Type Y characteristics as the axes. The current trajectory is 
determined and if a course change is desired, a preferred future trajectory is also defined. 
The trajectory is determined by the selection of statements related to: 



Organization mandate Monitoring R&D Projects 

Budget Sources Staff Deliverables 

Hiring Strategy Technology transfer 

Selecting R&D Projects Networking 

The statements have been prepared to describe the nature of organizatbns ranging fiTom 
pure Type X behaviour to pure Type Y behaviour in a set of five steps. (This is the only 
ProGrid tool that does not use an even number of steps in the ladder. For Module 1 , the 
levels in the ladder are not judgmental and the reason for ising an even number of steps 
does not apply). Depending on the statements selected, the trajectory on the grid is 
uniquely determined. An example of this is shown in the chart on the following page. 
This illustrates an extreme example in which an organization currently having 
dominantly Type Y characteristics wishes to shift to become a strong Type X 
organization. 
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R&D Trajectories 
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5.6 The Current and Desired Future Positions- Module 2 

Module 2 measures the extent to which an organization adheres to a set of practices, 
grouped into the nine cells shown in the following matrix. Column A includes the 
practices that are largely in the control of the research organization. The practices that 
involve interactions with the owTier and clients are in Column B and those that involve 
external interactions are in Column C. 
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These cells contain a number of perfonnance criteria, each having a set of four 
^'performance levels". The highest level describes what is currently considered as the 
maximum level of achievement possible for that fector, based on the judgement of 
knowledgeable R&D managers who have contributed to the development of the 
methodology. The highest level does not necessarily represent a desired level for an 
organization. The highest level may be inconsistent with die mandate and purpose of an 
organization. 

Based on the selections made by the organization, a position is determined on the 
trajectory defined in Module 1, and the distance traveled to date and the distance to the 
preferred fixture state is also defined (see pre\dous chart). 

Onc€ the trajectories and current and desired fijture positions are defmed, examination of 
the individual statements that have been selected will lead to an action plan designed to 
change course and achieve the target position. It is usefiil to calculate the "scores" 
obtained in each of the nine cells in the original matrix, as shown in the followdng profile 
chart. This assists in developing an action plan to achieve the desired fiiture state. 



Performance Profile 




42 



Copyright J999ProGrid Ventures Inc. & Clement W, Bowman Consulting Inc. 



5,7 Case Studies 



Approximately thirty R&D organizations have participated in projects that have included 
the use of the above "Best Practices" assessment process. Five types of organizations 
have been involved. 



1 . Private Sector- Single Company 

2. Private Sector- Consortia 

3. Government Laboratories- Industrial Sector Related 

4. Government Laboratories - General Industrial Support 

5. Government Laboratories- Science/Technology Focused 



Although the results are proprietary to the specific organizations involved, one major 
conclusion was reached that can be more broadly shared. This is illustrated in the chart 
below. Many government R&D organizations currently position themselves in a zone 
where they put more emphasis on the generation of intellectual capital than on its use. 
Individual companies place themselves more strongly in a zone v\^ere the use of 
intellectual capital is the more dominant focus. 



Performance Grid 




2 3 4 5 6 7 8 

Type X: Uses Intellectual Capital 



Neither of these entities is entirely satisfied with its current position. Many companies 
have coalesced around R&D consortia in order to achieve a better balance in short and 
long term activities, as shown by the Industry Consortia position. Government 
laboratories on the other hand are attempting to become more responsive to the needs of 
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clients and identify as a target a position closer to the current position of indi\idual 
companies. It is unlikely that government laboratories can put in place the necessary 
tight linkages among a broad range of clients to achieve this position. More importantly 
it raises the question as to whether they will so weaken their position for generating 
intellectual capital that they will cease to perform the mission for which they were 
established. 



The chart below displays the profile of typical Type X organizations, with the practices 
groined by indi\ddual matrix cells. 

Type XI- A Type X organization with relatively strong ratings across most of the profile, 
in particular linkages to its clients. Areas identified for attention are improved 
performance measurement and better access to global technology. Linkages to the 
external non-client community are not seen as a key factor for this type of organizatioa 

Type X2' A Type X organization with relatively poor links to its clients. None of the 
practice areas are strong, indicating that this organization has not yet undertaken a serious 
upgrade in its operations in response to competitive pressures. This position would likely 
not be sustainable in the long term. 



Type X Performance Profile 
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The chart below displays the profile of typical organizations that have both Type X and 
type Y characteristics, with the practices grouped by indi\ddual matrix cells. 

Type XYl- A mixed Type X and Y organization with sohd strength across the profile. 
The organization has been able to put in place the longer-term programs characteristic of 
Type Y organizations, without detracting fix)m its client linkages. TTiis is a typical 
situation for industry consortia. Linkages outside the client community are seen as 
important, for example in representing the interests of the members of the consortia to the 
government and the general public. 

Type XY2- A mixed Type X and Y organization wth strong links to the global R&D 
community active in its field of research. However, if flie organization is operating 
largely with client derived revenues, as many were in the studies that have been carried 
out, the poorly developed links with these clients cast doubt on the continued flow of 
these revenues. This position is probably not sustainable in the long term. 



Type X/Y Performance Profile 
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5.8 Alternative Evaluation Matrices 



The set- piece ProGrid-RO tool described above can be modified to fit individual user 
needs. For example, if an organization wishes to be more explicit in its commercial 
impact, the evaluation matrix can be amended to provide more direct evidence of this 
impact. 

In the example shown below, a new cell measuring the level of corporate impact has been 
added. The thrust of several other cells has been adjusted to adapt to this change. 

With the appropriate Language Ladder changes, grid and profile bar charts can be 
produced analogous to those presented in the previous sections. 
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5.9 What Has Been Learned? 

1. Industrial R&D organizations have evolved through a series of phases during the past 
few decades, involving a transition from a ''corporate'' status remote from day to day 
company operations to a position of close linkage to specific business cost centres. 
The final stage in this transformation is the acceptance of the R&D entity as a full 
partner in developing the strategic intent of the company. 

2. The above evolution has paralleled a realization that the intellectual capital of a 
company has become more important than its physical assets. The R&D organization 
has become a key participant in the generation and use of a con^any's intellectual 
capital. 

3. Intellectual Capital can be defined as the summation of intellectual property, codified 
knowledge, systems and processes and human resources. One approach for 
measuring intellectual capital is to examine the practices that an organization uses in 
managing these four asset categories. 
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4. In benchmaildng R&D organizations, the stage at which the organization has 
progressed and its specific mandate and mission must be clearly defined. The 
expectations of the owner and other principal stakeholders must be reflected in any 
benchmarking study. 

5. The ProGrid-RO methodology assesses the practices of an R&D organization, in 
comparison with leading pacesetter "best practices". The procedure measures the 
extent to which an organization adopts practices appropriate for the use of intellectual 
coital (Type X) and practices q)propriate for the generation of intellectual capital 
CFypeY). 

6. One of the major findings from the studies completed to date is the paradox of 
"^lifting missions". Many government R&D organizations are in transition from a 
focus on generating intellectual capital to a position where they are more closely 
linked with clients in the use of intellectual capital to solve short-term problems. 
Meanwhile, many companies, having made that same transition in the past tv/o 
decades are now realizing that this has been at the expense of protecting their long- 
term competitive position. These companies have now formed consortia in various 
sectors to collaborate in the renewal of their base of intellectual capital. 

7. Case studies have identified the practices that Type X and mixed Type XY 
organizations require to ensure long-term success. Organizations which lack strength 
in these key practices may not be sustainable, 

8. Those who have used the ProGrid-RO tool have found it to have a number of key 
advantages, which include: 

• Ensures that all key aspects have been addressed (open architecture) 

• Establishes a "Language Ladder'' process for assessing practices (essentially a 
"calibration" procedure) 

• Achieves consensus- provides input from all stakeholders 

• Tracks performance over time (separates "activities" and "progress") 

• Provides disciplined and reproducible results 

• Graphical and easily understood 

• A self- learning methodology, constantly improving 
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Chapter 6: Company Performance (ProGrid-CO) 



6.1 Introduction 

ProGrid-CO is a method for assessing the performance of a company with respect to its 
abiUty to achieve its vision and to meet its strategic plan. 

The methodology can be applied in several different ways, including: 

• By indi\adual managers to compare views and perspectives 

• By a management group to seek consensus on critical issues 

• At various levels in an organization to assess consistency in the application of various 
management practices 

• Among organizations to compare experiences and learnings 

The procedure evaluates a company v^th respect to two principal criteria: (a) 
organizational excellence and (b) the business opportunity. The inability of some 
companies to make the transformations required to ensure sustainability is a major cause 
for corporate failure. A powerful feature of the ProGrid-CO methodology is the 
identification of both the current and desired future states, and an accompanying set of 
actions. The nethodology has two key deliverables- (a) the location of the company on a 
grid that defines its status with respect to the two principal criteria noted above, and (b) a 
perfomiance profile which identifies specific strengths and weaknesses. These two 
outputs set the stage for the development of an effective strategy designed to improve 
future perfomiance. 

Development of the ProGrid-CO tool involved the standard five ProGrid steps: 

Step 1' Defme a series of evaluation criteria with respect to which the perfonnance is to 
be measured. 

Step 2' Select two of these as the overarching criteria that if both achieved yield the 
highest possible perfonnance level for tiie company. As noted above, organizational 
excellence and the business opportunity have proven to be effective overarching criteria. 
Construct an evaluation grid that has these criteria as the axes. 

Step 5- Organize the remaining criteria into a matrix, each cell of which contains related 
criteria. 

Step 4 Establish a series of quality or performance levels for each criteria in the matrix. 
In ProGrid terminology, these levels comprise the Language Ladder^*^ measurement 
system. 

Step 5- Construct a bar graph that compares ratings for each of the matrix cells. 
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6,2 Evaluation Criteria 



Discussions with various management consultants involved in assessing the perfomiance 
of companies generated a large number of key criteria that were indicative of both short 
and long term commercial success. 



Over time, the list has been modified to the following 58 criteria: 



Clarity of Vision 


Commitment to Goals 


Clarity of Mission 


Mission Focus ' 


Auditing Performance Against 


Board of Directors 


Vision/Mission 




Shareholders 


Organization Structure 


Public Relations 


Industry Relations 


Champion 


Senior Management 


Staff Depth and Breadth 


Succession Plan 


Team Culture 


Recruitment 


Training 


Recognition 


Safety Program 


Tracking Performance 


Organization Improvement 


Financial Plan 


Revenue Plan 


Capital Requirements 


Overall Financial Strength 


Return on Investment 


Production Cost Position 


Production Capacity 


Raw Material Cost 


Supplier Relationships 


Goodwill 


Marketing Plan 


Market Channels 


Customer Relationships 


Technical Support 


Patents/Copyrights 


Competitive Intellectual Property (IP) 


Trademarks 


Know-how 


Intellectual Capital Management 


Business Plan 


Plan Use and Validation 


Operating Goals 


Strategic Partners 


Business Threats 


Product Offering 


Margins 


Product Obsolescence 


Product Improvements 


Quality 


Market Size 


Market Penetration 


Geographic Reach 


Regulatory Environment 


Competitive Situation 


Competitive Intellectual Property Position 


Competitive Dominating Business Factors 


Competitive Financial Strength 
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6.3 Overarching Criteria 

Most of the above criteria govern either or both the excellence of the organization or the 
potential of the business opportunity. 

This led to the selection of the following two overarching criteria: 

The Organization 

The Business Opportunity 
as the axes of the evaluation grid as shown in the following chart. 




6.4 The Evaluation Matrix 

The above criteria were grouped into clusters of related factors in the following matrix. 

The columns in this matrix represent a progression from factors that largely influence the 
organizational strength of the company in Column A, to those that largely influence the 
business potential in Column C and those that are necessary factors to enable the 
organization to capture the business opportunity in Column B, the connecting factors. 
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■■^^^^ 




The Vision/Mission 

• Clarity of Vision 

• Commitment to Goals 

• Clarity of Mission 

• Mission Focus 

• Auditing Performance 
Against Vision/Mission 


The Financials 

• Financial Plan 

• Revenue Plan 

• Capital Requirements 

• Overall Financial Strength 

• Return on Investment 


The Plan 

• Business Plan 

• Plan Use and Validation 

• Operating Goals 

• Strategic Partners 

• Business Threats 


The Leadership 

• Board of Directors 

• Shareholders 

• Organization Structure 

• Public Relations 

• Industry Relations 


Production Capability 

• Production Cost Position 

• Production Capacity 

• Raw Material Cost 

• Supplier Relationships 


The Products 

• Product Offering 

• Margins 

• Product Obsolescence 

• Product Improvements 

• Quality 


The Team 

• Champion 

• Senior Management 

• Staff Depth and Breadth 

• Succession Plan 

• Team Culture 


Marketing System 

• Goodwill 

• Marketing Plan 

• Market Channels 

• Customer Relationships 

• Technical Support 


The Market Opportunity 

• Market Size 

• Market Penetration 

• Geographic Reach 

• Regulatory Environment 


Processes and Practices 

• Recruitment 

• Training 

• Recognition 

• Safety Program 

• Tracking Performance 

• Organization Improvement 


Intellectual Capital (IC) 

• Patents/Copyrights 

• Competitive Intellectual 
Property (IP) 

• Trademarks 

• Know-how 

• Intellectual Capital 
Management 


The Competition 

• Competitive Situation 

• Intellectual Property 
Position 

• Dominating Business 
Factors 

• Financial Strength 



6,5 The Language Ladder 

This is the critical step that estabUshes the metrics of the ProGrid process. Through 
experience, it has been found that a set of four steps in the Language Ladder work well. 
The choice of an even number is important in that it reduces the tendency for evaluators 
who use the ProGrid methodology to migrate to the middle "neutral" position. 

The following are two examples of Language Ladder statement sets that are used in the 
current ProGrid-CO procedure. 
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The Board of Directors 

A. The organization has not yet been incorporated into the form necessary to conduct 
an ongoing successful business. 

B. The organization has been incorporated and a Board of Directors established to 
meet legal requirements. The Board has not yet had any active role in the 
direction of the otganizatioa 

C. The organization has been incorporated and the members of the Board of 
Directors are well known in their fields of experience and have been able to 
provide guidance at legislatively required meetings of the Board. 

D. The organization has an effective Board of Directors with a good balance of 
internal and external representation. Members of the Board actively participate in 
the aflPairs of the organization through frequent Board and Conunittee meetings. 

Dominating Business Factors 

With respect to the products/processes/services provided by this concept: 

A. The competition has a dominant position in one or more key business factor(s), 
such as raw material cost, locked in contracts or restricted distribution channels, 
which will seriously restrict the entry of third parties. 

B. The competition has in the past had a dominant position in one or more key 
business factor(s) but these can be effectively overcome by tie organization, as 
defined in the business plan 

C. No competitor has a significant dominating business factor that will adversely 
affect the entry of third parties. 

D. Competitors will be deterred by the organization's dominant position in one or 
more key business factors, such as raw material cost, locked in contracts or 
restricted distribution channel. 

6.6 Evaluation Grid 

Once a company has been evaluated using the ProGrid methodology, its position can be 
represented on the following grid, depending on the level of maturity of die 
organization's stmcture and practices and the potential of the business opportunity. 

Both the current and the desired future position can be assessed. Once the long term 
target has been established, it is useful to subdivide the changes required into various 
stages, which could represent time targets as shown in this grid. Alternatively, targets that 
represent the contribution of individual departments in the organization could be defined. 

This type of chart can be effective in tracking changes in the organization as a new 
strategic plan is implemented. 



52 



Copyright 1999 ProGrid Ventures Inc. & Clement IV. Bowman Consulting Inc. 



Company Performance Grid 




0 -I 1 1 1 1 — 1 1 1 1 1 

0 5 10 

The Business Opportunity 



6,7 Interpreting the Grid Position 



The position of an organization on the grid will be heavily dependent in its stage of 
development and past history, which can be captured in the following chart. 







Stage 






Embryonic 


Emerging^ 


Growth 


Stable 


Decline 


Service 
Companies 


Information 












Evaluation 












Financial 












Other Service 












Wealth 
Generators 


Manufacturer 












Marketing 












Integrated 
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Whether it is a direct wealth generator through the production of tradeable goods and 
services or whether it is has a service- supporting role will be important factors to 
consider in the analysis. The chart is a asefiil mefliod of classifying a company and 
understanding the areas where particular strengths and weaknesses are critical. 

6.8 The Profile Chart 

In addition to the grid position, a bar chart of the strengths of individual cells in the 
evaluation matrix is an important output of the ProGrid methodology. This helps to 
identify the particular strengths and weaknesses of flie company. 

In the example shown, the company has near term work to do on the visionAnission and 
then to translate this into an effective Plan. There is also near term objectives related to 
some of the internal processes and practices and to enhancing the Marketing System. 
Longer-term goals include increasing production capacity and strengthening the slate of 
products. 



Performance Profile 



100 
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Chapter 7: University Practices (ProGrid-UN) 



7.1 Introduction 

A university can undertake a self- assessment using ProGrid-UN (on a departmental 
and/or a total university basis) wth respect to its principal missions of teaching and 
research excellence. The method differentiates universities that aspire to be leaders in the 
quality of teaching, that have research as their principal mission, or fliat seek to be 
pacesetters in both teaching and research excellence. 

A powerful feature of this methodology is the identification of both the current position 
and the desired future state of the university, and an accompanying set of actions. There 
is no "right" position for all universities. The position of any particular university flows 
from its vision and mission statements. 

The development of the ProGrid-UN methodology arose out of a University of Toronto 
initiative in 1994 to develop a new strategic plan. The objective of the plan was to place 
the University in the top quartile of North American universities in both teaching and 
research excellence. Many of the supporting criteria had already been identified in the 
strategic plan. 

Development of the ProGrid-UN tool involved the standard five ProGrid steps: 

Step 7- Define a series of evaluation criteria with respect to which the performance is to 
be measured. 

Step 2- Select two of these as independent overarching criteria that if both achieved yield 
the highest possible performance of the university. As noted above, teaching and 
research excellence, have been identified by the overarching criteria by some universities. 
Constmct an evaluation grid that has these two criteria as the axes. 

Step 3- Organize the remaining criteria into a matrix, each cell of which contains related 
criteria. 

Step 4- Establish a series of quality or performance levels for each criteria in the matrix. 
In ProGrid terminology, these levels comprise the Language Ladder™ measurement 
system. 

Step 5- Constmct a bar graph that compares the ratings of each of the matrix cells. 

7.2 Evaluation Matrix 

The strategic plans of several universities were reviewed in establishing the following 
matrix as the embodiment of their vision and missioa The industry and the community 
hnkages are receiving increased attention, coincident with the increasing interest of 
society in the role of universities in knowledge-intensive economies. 
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Research 


Links To Industry And The 
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7.3 Four-Quadrant Grid 

Using the above two overarching criteria as grid axes results in the following evaluation 
grid The four quadrants in this grid show different degrees of specialization, ranging 
from research institutes whose teaching role would be limited to specific post graduate 
programs to institutions whose primary emphasis is leading edge teaching practices with 
only a minimum research focus. 



University Positioning Grid 
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Teaching Excellence 
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A University of Montreal staff member has suggested that a more fundamental 
description of the axes would be Generation of Information and Conservation of 
Information. These might better describe the purpose of research and teaching, and have 
some analogy to the generation of inteUectual capital and use of intellectual capital 
described in Chapter 5. 

7.4 Evaluation Criteria 

With the help of a staff member of Curtin University in Western Australia, the following 
criteria were established within each cell of the Evaluation Matrix. 



Under Column A- Undergraduate Programs 



lillillii®«^wt^/ 




lliiliiiii^ilpif^^ 


7 

Relevance of Cutriculu 


1 

Entrance Qualifications 


1 

Lreaennals oj iStaJj 


2 

{^ohp^ivpnPKK of thp 
Curricula 


2 

Choice of Institution 


2 


3 

Integration Among 
Courses 


3 

Student Assessment of 
Staff 


3 

Promotion Ladder 


4 

Curriculum Delivery 


4 

Student Involvement in 
Governance 


4 

Hiring Practices 


5 

Curriculum Accessibility 


5 

Student Honours 


5 

Collegiality 


6 

Laboratory Support 


6 

Graduate Follow-up 


6 

Involvement in the 
Profession 


7 

Adjunct Professors 


7 

Student Support Program 


7 

Professional 
Development 






8 

Governance 
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Under Column B- Graduate and Research Programs 
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1 

Overall Program Quality 


1 

Quality of research 


] 

Facilities 


2 

UKuaUClW L.OllfS€S 


2 

Kesearcn L/uipui 


2 

oKJiiea iraoes 


J 

Thesis 
Programs 


7 

Research Focus 


Computer Services 


4 

Balance of 
Local/Regional/Foreign 
Graduates 


4 

Funding Continuity 


4 

Information Services 


5 

External Reviews 


5 

Discretionary Funding 




6 

Retention Time 







Under Column C- External Linkages 





Linkk to^Qth^N^ 
.;.=!• Jhstiiutions 




1 

Vision and Mission 
Alignment 


1 

Visits and Staff 
Exchanges 


J 

Advisory Committees 


2 

Evaluation Processes 


2 

Joint Projects 


2 

Cooperative Projects 


3 

Collaboration in Courses 


3 

Twinning Arrangements 


3 

Courses for Industry 


4 

Interdisciplinary Programs 


4 

Equipment Sharing 


4 

Technology Transfer 


5 

Joint Thesis Supervision 




5 

Considting Services 






6 

Satisfaction Surveys 






7 

S&T Message 
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7.5 Language Ladder 

For each of the above criteria, a set of four Language Ladder statements was established 
to provide the metrics for the evaluation system. The choice of an even number is 
important in that it reduces the tendency for evaluators who use the ProGrid methodology 
to migrate to the middle ''neutral" position. 

The following are two examples of Language Ladder statement sets that are used in the 
current ProGrid- UN procedure. These two examples are designed to be applied at a 
Departmental Level, but analogous statements can be developed at the University- wide 
level. 

Relevance of Curricula 

A. The curriculum was established more than ten years ago, largely on a discipline 
basis. Only minor changes to the curriculum have since been made. 

B. The curriculum has had an internal review during the past five years and tes been 
upgraded based on the findings of diat review. 

G. The curriculum has been reviewed by an external panel during the past five years 

and has been upgraded as a result of that review. 
D. The curriculum is highly relevant to today*s business and social needs and has 

been developed in close consultation with all major stakeholders, including 

industry, graduates and related research bodies. The curriculum is reviewed and 

upgraded on an ongoing basis. 



Advisory Committees 

A. The Department has not established an Advisory Committee to seek advice from 
major stakeholders. 

B. The Department has established an Advisory Committee that has one or two 
industrial members. The committee meets infrequently and has not yet had any 
significant influence on the affairs of the Department. 

C. The Department has established an Advisory Committee having strong industrial 
representation that meets on an irregular basis. Some of the recommendations 
from the committee have been adopted, but the committee has not yet had a major 
impact on the affairs of the Department. 

D. The Department has established an active Advisory Committee having strong 
industrial representation that meets at least once a year. The recommendations of 
this committee have been adopted and contributed to the success of the 
Department. Althou^ the Committee is Advisory, it is considered to be more like 
a Board of Directors by its members. 



Copyright 1999 ProGrid t^entures fnc. <fc Clement W. Bowman Consulting Inc. 



59 



7.6 Carrying out the Assessment 



The ProGrid-UN assessment would be best carried out as part of an overall strategic 
planning exercise. Specific components could be undertaken by various stakeholders, 
from the Board of Governors to representatives of the student body. At this time, there 
has been no detailed application of the ProGrid-UN methodology since the fundamental 
framework was developed. 



7.7 The Evaluation Grid 

Once a university has been evaluated using the ProGrid methodology, its position can be 
represented on the grid shown below. 

Both the current and the desired ftiture position can be assessed. Once the long term 
target has been established, it is useful to S'ubdivide the changes required into various 
stages, which could represent time targets as shown in this grid This type of chart can be 
effective in tracking changes in the university as a new strategic plan is implemented. 
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Teaching Excellence 
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7.8 The Profile Chart 



Hie following profile chart provides more in-depth analysis of the affect of the new 
strategic plan and would be used in tracking performance. 

The hypothetical example shown in these two charts is a university planning a near-term 
enhancement in its research focus, through enlarged graduate programs and the addition 
of leading edge researchers. Longer-term there will be more emphasis on reaching 
pacesetter teaching performance through techniques such as remote teaching services. 



The University Profile 



100 
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Chapter 8; Non-Technical Applications 



8*1 Introduction 

The concept of assessing intangibles using ProGrid methodology has application outside 
purely scientific and technical areas. The key factor is whether there are two overarching 
criteria that are independent of one another, i.e.. both criteria can be met independent of 
the others. 

One specific non-technical application discussed in this section is the measurement of the 
contribution of staff to the vision, mission and goals of the organization, for performance 
evaluation and/or compensation. 

8.2 Personnel Performance Assessment (ProGrid-HR) 

An illustrative example is that of consulting firms providing services to clients. These 
could, for example, be engineering or legal services. 

Following the standard ProGrid approach, the criteria on v/hich individual contributions 
are to be measured ^should first be identified. This must include every possible 
contribution that is valued by the organization. Unlike many of the other ProGrid 
applications, the objective is not to expect contributions across all criteria but to 
recognize high contributions in what may be in only a very few criteria, which may be 
recurring or one-tiine events, e.g. "spikes" of performance. 

Once the criteria have been estabhshed, they should be organized in a typical ProGrid 
evaluation matrix in the form shown below. This example pertained to a consulting 
organization where both the development of tools to meet client needs and die woric of 
delivering services to the clients were recognized 



Wm:-^::llt€iBasic^ Topis mJ^ 


r. The Support liPljif 




Acquired Expertise 


Administrative Support 


New Business Secured 


Development of Advanced 
Processes/Products 


General Marketing Support 


Participation in Projects 


Knowledge Integration 


Client Support 


Strategic Impact 



This suggested that the two overarching performance criteria were "Building the Basics" 
and "Creating Value". These were selected as the axes of the performance grid as shown 
below. 
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83 Language Ladder Statements 

Examples of two Language Ladder statement sets are provided below: 
Development of Advanced Processes/Products 

During the past assessment period, the staff member has: 

A. Not been involved to any significant extent in the development of new advanced 
processes/products/services for the organization. 

B. Has participated in the development and testing of at least one new advanced 
process/product/services. 

C. Has played an important role in the development of new advanced 
processes/products/services, including conducting effective beta tests in a 
commercially relevant setting. 

D. Has played a leadership role in the conception, construction and market testing of 
new advanced processes/products/services that fit well with known or prospective 
maricet needs. 

Strategic Impact 

During the assessment period, the staff member has: 

A. Had little or no impact in formulating corporate strategies or achieving key 
strategic objectives. 

B. Had some involvement in formulating corporate strategies and has supported 
some aspects of achieving key strategic objectives. 
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C. Been active in and has made significant contributions to certain aspects of 
corporate strategy formulation and the achievement of key strategic objectives. 

D. Been a leader in both the formulation of strategies for the corporation and in 
achieving key strategic objectives. 

8.4 Using the Procedure 

The procedure can best be employed as a part of ongoing performance evaluation. In 
keeping with modem human resource practices, an initial self- assessment can be candied 
out using a prescribed form in which the employee selects his/her performance level in 
each cell of the matrix and provides a short justification for the selection. This can then 
be reviewed by a group of associates who may be in a supervisory, coUegial, or 
supervised position with respect to the employee. This proAades a group consensus by 
those who know most intimately the work of the employee. 

This assessment would then make an effective document for the employee and his/her 
supervisor to use in the regular performance appraisal 

8.5 Contribution Charts 

The two charts on the following page illustrate examples of the output from the 
assessment. The Compensation Grid shows the self- assessment by the staff member and 
the average assessment of the reviewers. The indi\ddual reviewer assessment can also be 
shown, if desired. 

The descriptions of the zones in the Contribution Grid are illustrative only and would 
need to be customized to suit the practices and culture of the organization. 

The profile chart identifies the areas of contribution and where there are differences 
between the staff member's ratings and those of the review team. In this example the 
staff member has made major contributions in maiketing and client support and has 
brought in a significant amount of new business. Contributions to new products and 
services were not as dominant. 
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Creating Value 



Contribution Profile 
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8.6 Other Applications 



There are many other applications where a ProG rid -style tool could be effectively 
employed. Any selection processes where there are two independent criteria to balance 
are candidate areas. Possibilities include: 

• Selecting sites for athletic events 

• Selecting artistic awards 

• Awarding charitable grants 

• Recognizing individual achievements 

• Selecting career paths 

• Selecting entrance applications 

In applying ProGrid to these application, it would be important not to prejudge the 
independent overarching criteria but to proceed though a process where all the values, 
priorities and expectations of the selecting agency are first defined and grouped, as 
outlined in the previous chapters of this manuscript. 

There are many areas to explore in the hidden part of the iceberg! 
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